Skip to content

DM-54894: Add new utilities from analysis notebooks and update existing plots#64

Open
isullivan wants to merge 30 commits into
mainfrom
tickets/DM-54894
Open

DM-54894: Add new utilities from analysis notebooks and update existing plots#64
isullivan wants to merge 30 commits into
mainfrom
tickets/DM-54894

Conversation

@isullivan

Copy link
Copy Markdown
Contributor

No description provided.

@BrunoSanchez BrunoSanchez self-requested a review May 19, 2026 12:29
@isullivan isullivan force-pushed the tickets/DM-54894 branch 8 times, most recently from 301f00f to 2aaff4d Compare May 21, 2026 19:54
isullivan added 20 commits June 12, 2026 16:28
Also clean up the remaining docstrings
Read integer columns through the SDM schema so nullable BIGINTs round-trip
exactly instead of being silently coerced to float64 by pd.read_sql_query.
Fall back from validityEndMjdTai to the legacy validityEnd name for
older APDB fixtures.
The metadata rows and flag-tag overlays are now declarative tables driven
by a shared loop, instead of a long sequence of inline fig.text calls.
A new heights kwarg lets subclasses with extra panels position the rows
themselves.
New compare module: match_catalogs cross-matches two DiaSource frames
within optional group columns, flux_residuals computes per-band/per-flux
residual statistics for matched pairs, and match_to_truth attaches the
nearest truth row to each detection.
display_images is rewritten around a shared overlay-collection helper:
catalogs are loaded once, color-coded by flag group when requested, and
the same overlays are reused for every frame. New display_images_ab puts
two butlers side-by-side, and extract_timestamped_messages flattens an
LSST JSON log.
Three helpers: lightcurve plots a per-band psfFlux history for a single
diaObject (with optional forced-photometry overlay), cutout_grid lays out
science/template/difference cutouts for many sources in a mosaic, and
summarize_run produces a per-visit summary of an APDB run.
Extends PlotImageSubtractionCutoutsTask with a lightcurve panel beneath
the science/template/difference cutouts. The panel shows the diaSource
history for the associated diaObject and overlays forced-photometry
measurements for visits without a detection.
Walks the per-(visit, detector) DiaSource/DiaObject/DiaForcedSource
datasets that DiaPipelineTask writes alongside its APDB inserts and
concatenates them into single DataFrames matching the APDB SDM schema.
Useful for QA without round-tripping through a live APDB.
image_diff_stats reports median, MAD, stdev, kurtosis, and mask-plane
fractions for a single difference image. pixel_compare aligns two
difference images on the same dataId and returns their pixel difference,
ratio, mask XOR, and summary statistics.
subtraction_quality_report consumes a per-detector
*_spatiallySampledMetrics table and prints percentile summaries plus a
three-panel diagnostic figure for diffim_chi2PerPix,
psfMatchingKernel_residualNorm, and dipole_density.
Walks every <task>_metadata dataset in a butler collection, extracts
timing fields via QuantumResourceUsage.from_task_metadata, applies a
per-task threshold (keep the whole task if any quantum exceeds it), and
returns a tidy DataFrame with an optional box plot.
`find_objects_sharing_sources` walks the (diaSource, run-1 diaObject,
run-2 diaObject) graph to return the full connected cluster reachable
from a starting diaObjectId. `classify_association_clusters` enumerates
every such cluster across two runs and labels each as matched, split,
merged, or tangled.
`plot_cutouts_with_object_markers` renders per-diaSource cutouts with
color-coded diaObject markers and "other diaSource" overlays.
`plot_objects_sharing_sources` is a two-column wrapper that compares
the run-1 and run-2 views of the cluster reachable from a starting
diaObjectId.
In plot_objects_sharing_sources, source markers now take the palette
color of the diaObject they belong to instead of a single shared color.
The right panel maps sources to their run-2 owner so the same source can
read differently across the two runs.
… visible.

Also ensure that other markers are drawn in pipeline order, so that the last version is the one that's visible.

@BrunoSanchez BrunoSanchez left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have read all changes, I am on my way now to test some of these in a notebook with some Rubin alert data, or CI package.
I have run your notebooks but they don't show all the functionality added here.
I approve the changes, but will keep commenting. I think it is good to merge, but building more examples can be useful, and help also test this a bit more.

Comment thread python/lsst/analysis/ap/__init__.py Outdated
from .spatiallySampledMetricsQA import *
from .plotUtils import *
from .taskRuntimes import *
# NOTE: do not import from nb_utils in this file, as it depends on packages

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about this? You are adding an from .nb_utils import *

return unique1, unique2, matched


def find_objects_sharing_sources(diaObjectId, sources1, sources2,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't yet tested this, but it is a very interesting functionality.

Comment on lines +49 to +57
BAND_COLORS = {
"u": "C4",
"g": "C2",
"r": "C1",
"i": "C3",
"z": "C5",
"y": "k",
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not loading the lsst official color palette?

The existing erase call only erased the image, not any markers drawn on top of it.
@isullivan isullivan force-pushed the tickets/DM-54894 branch 3 times, most recently from 361f82f to 254eaf1 Compare July 14, 2026 17:49
@isullivan isullivan force-pushed the tickets/DM-54894 branch 4 times, most recently from 9793747 to 47113a4 Compare July 16, 2026 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants